Computer-implemented method for generating an output video from multiple video sources

ABSTRACT

An output video is created by at least two cameras recording respective source videos, each having multiple video frames containing video objects imaged by the cameras corresponding to multiple instances of one or more respective source objects traversing the site. Output video objects having a new start display time are computed such that a total duration of display times of all video objects from all source videos is shorter than a cumulative duration of the source videos. The output video objects or graphical representations thereof are rendered at new display times over a background image such that (i) instances imaged by different cameras at different times are represented simultaneously; (ii) at least two output video objects originating from a common camera have different relative display times to their respective source objects; and (iii) in at least one location there are represented instances imaged by two different cameras.

RELATED APPLICATION

This application claims benefit of provisional application Ser. No.62/754,904 filed Nov. 2, 2018 whose contents are incorporated herein byreference.

FIELD OF THE INVENTION

This invention relates to video synopsis.

PRIOR ART

Prior art references considered to be relevant as a background to theinvention are listed below and their contents are incorporated herein byreference. Additional references are mentioned in the above-referencedU.S. Ser. No. 62/754,904 and its contents are incorporated herein byreference. Acknowledgement of the references herein is not to beinferred as meaning that these are in any way relevant to thepatentability of the invention disclosed herein. Each reference isidentified by a number enclosed in square brackets and accordingly theprior art will be referred to throughout the specification by numbersenclosed in square brackets.

-   [1] A. Rav-Acha, Y. Pritch, and S. Peleg, “Making a long video    short: Dynamic video synopsis,” in CVPR, 2006, pp. 435-441.-   [2] Y. Pritch, A. Rav-Acha, and S. Peleg, “Nonchronological video    synopsis and indexing,” IEEE Trans. PAMI, vol. 30, no. 11, pp.    1971-1984, 2008.-   [3] S. Peleg and A. Rav-Acha, “Method and system for producing a    video synopsis,” U.S. Pat. No. 8,102,406, 2012.-   [4] S. Peleg, Y. Pritch, A. Rav-Acha, and A. Gutman, “Method and    system for video indexing and video synopsis,” U.S. Pat. No.    8,311,277, 2012.-   [5] S. Peleg, Y. Pritch, S. Ratovitch, and A. Hendel, “Method and    system for producing a video synopsis using clustering,” U.S. Pat.    No. 8,949,235, 2015.-   [6] E. Richardson and S. Peleg, “Method and system for producing    relevance sorted video summary,” U.S. Pat. No. 9,877,086, 2018.-   [7] Y. Hoshen and S. Peleg, “Live video synopsis for multiple    cameras,” in ICIP, 2015, pp. 212-216.-   [8] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards    real-time object detection with region proposal networks,” in    Advances in neural information processing systems, 2015, pp. 91-99.-   [9] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu,    and A. C. Berg, “SSD: Single shot multibox detector,” in European    conference on computer vision. Springer, 2016, pp. 21-37.-   [10] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,”    in Computer Vision (ICCV), 2017 IEEE International Conference on.    IEEE, 2017, pp. 2980-2988.-   [11] W. Luo, J. Xing, A. Milan, X. Zhang, W. Liu, X. Zhao, and T.-K.    Kim, “Multiple object tracking: A literature review,” arXiv preprint    arXiv: 1409.7618, 2014.-   [12] R. Hartley and A. Zisserman, Multiple view geometry in computer    vision. Cambridge university press, 2003.-   [13] H. S. Sawhney, A. Arpa, R. Kumar, S. Samarasekera, M.    Aggarwal, S. Hsu, D. Nister, and K. Hanna, “Video flashlights: real    time rendering of multiple videos for immersive model    visualization,” in ACM International Conference Proceeding Series,    vol. 28, 2002, pp. 157-168.-   [14] U.S. Pat. No. 8,063,936, “Modular immersive surveillance    processing system and method”-   [15] K. Huang, C. Lijun, and W. Chen, “Large-range-first    cross-camera visual target re-identification method,” U.S. Pat. No.    9,996,939, 2018.-   [16] W. Ayedi, H. Snoussi, and M. Abid, “A fast multi-scale    covariance descriptor for object re-identification,” Pattern    Recognition Letters, vol. 33, no. 14, pp. 1902-1907, 2012.-   [17] A. Bedagkar-Gala and S. K. Shah, “A survey of approaches and    trends in person re-identification,” Image and Vision Computing,    vol. 32, no. 4, pp. 270-286, 2014.-   [18] X. Liu, W. Liu, H. Ma, and H. Fu, “Large-scale vehicle    re-identification in urban surveillance videos,” in Multimedia and    Expo (ICME), 2016 IEEE International Conference on. IEEE, 2016, pp.    1-6.

BACKGROUND OF THE INVENTION

Reviewing scene activity by watching video clips from surveillancecameras is boring and time consuming. In many cases the activity issparse, and time compression can be achieved by extracting the movingobjects from the original video and reviewing only these objects. VideoSynopsis [1, 2, 3, 4] makes the review faster by re-arranging theextracted objects in time so that high temporal compression is achieved.The re-arranged objects are usually displayed on a synthetic backgroundimage that is learned statistically from the original video.

While allowing fast review of long videos with sparse activity, VideoSynopsis has several significant limitations: (i) Video Synopsis isbased on the assumption that the camera is static; (ii) Video Synopsisis limited to the camera viewpoint; and (iii) the extracted objectsshould be segmented accurately to avoid displaying artifacts whilestitching them on the background image. The segmentation may beperformed incorrectly in many scenarios such as mutually occludingobjects, hardly visible objects, etc.

Video Synopsis

Video Synopsis as described in [1, 2, 3, 4] relates to a video recordedby a single stationary camera. Moving objects are first extracted fromthe original video, and then temporally re-arranged and displayed on theoriginal background. The result is a shorter video, showing all theactivity in the scene in shorter time.

Extensions of Video Synopsis include presentation of clustered objects[5] and ordering the displayed objects by their relevance to apredefined objective function [6]. In [7] an attempt is made to usesynopsis in a multi-camera case, by arranging a synopsis of the objectsin one camera based on objects that appear in another camera.

All the above-mentioned works generate video clips which display theextracted objects in their original surrounding, i.e., on a backgroundimage learned from the same camera in which the objects appeared. Otherdisplay modes, such as displaying icons or using background unrelated tothe original camera, have not been used.

Object Extraction

Spatio-temporal rearrangement of the scene objects, as done by VideoSynopsis, requires a preliminary stage in which the objects of interestare detected and tracked in the original video. In [1, 3], the scenebackground model is learned statistically, and moving objects areextracted by their difference from the background model. Such abackground subtraction method is applicable as long as the recordingcamera is static. An alternative method is executing an object detectorsuch as Faster-RCNN [8] or SSD [9] over the input video frames,providing the bounding box locations of the scene objects. Pixel levelobject masks can be computed using instance segmentation methods such asMask-RCNN [10]. These methods are applicable also for video framesrecorded by a moving camera.

Multiple object tracking methods such as reviewed in [11] connect thedetections of each individual object in different video frames, based onappearance and motion similarity. This provides the trajectories of thedifferent scene objects.

Any of the existing methods for background subtraction, moving objectsegmentation, and objects tracking, known by people skilled in the art,is possible. The result after applying these methods is a “tube”representing a single object: a sequence of locations of this object insuccessive frames, from its earliest appearance to its last appearance.

Camera Parameters

For each camera there are associated internal and external parameters.Internal parameters include optical parameters such as focal length andlens distortion. External camera parameters include 3D pose of thecamera (i.e. pan, tilt, roll angles) and its 3D spatial location.Estimation of internal and external camera parameters is described in[12].

It is also possible to calibrate the camera with a 2D or 3D surfacewithout estimating all the camera parameters. Calibration of a planarsurface in two views can be done by matching at least four correspondingpoints [12]. This calibration can be used for mapping objecttrajectories from the image plane into 2D reference models such as a mapor a diagram.

The 3D pose of a camera relative to a ground surface can be estimatedfrom the angles measured by an accelerometer attached to the camera. In[13, 14], a method is proposed to calibrate a camera with a reference 3Dmodel using a 3D pose estimation module. For fixed cameras, the pose isestimated once. For moving cameras (e.g. Pan/Tilt/Zoom and aerialcameras), pose can be estimated every frame. This allows the objecttrajectories inside the 3D model to be located.

Object Matching

Multiple instances of an object can be recognized using severalapproaches. If the object is visible simultaneously by two calibratedcameras, matching can be done by determining whether the two instancesoccupy the same space at the same time. In other cases, pairs of videoobjects can be matched by their similarity, such as appearance and/ormotion similarity. This technology is called object re-identification[15, 16, 17, 18].

SUMMARY OF THE INVENTION

In accordance with a broad aspect of the invention, there is provided acomputer-implemented method for generating an output video, the methodcomprising:

obtaining respective source videos recorded by at least two cameras in asite, each of said source videos comprising multiple video framescontaining video objects imaged by said cameras, said video objectscorresponding to multiple instances of one or more respective sourceobjects;

obtaining for detected video objects in each source video, respectivetracks containing locations of the respective video objects in the videoframes;

for at least some of said video objects computing output video objectshaving a new start display time of each video object

selecting a background image on which to render the output videoobjects; and

generating an output video by rendering the output video objects orgraphical representations thereof at their new display times over theselected background image such that:

-   -   (i) instances imaged by different cameras at different times are        represented simultaneously in the output video;    -   (ii) at least two output video objects originating from a common        camera have different relative display times to their respective        source objects; and    -   (iii) there exists at least one location in the output video        wherein there are represented instances imaged by two different        cameras.

Although the invention is particularly applicable to the case where asite is recorded by multiple cameras thus producing multiple videosequences, it also contemplates the case in which a site is viewed by asingle camera that records a single video sequence, as long as thedetected object trajectories are projected onto a reference model, and avideo synopsis is generated from the representation in the model.

For the sake of clarity, it should be noted that the term “sourceobjects” refers to physical objects that are imaged by the cameras,which then create “video objects” or, more simply, “objects”. The videoobjects are video frames or portions of video frames that depict atleast part of the source objects. For example, a video camera may imagea person's face so that successive frames of a video sequence containvideo objects depicting the respective images. Video objects in discretevideo sequences can be matched using known techniques so as to determinewhether or not they relate to the same source object. In someembodiments this is done to establish that video objects imaged bydifferent cameras that image non-overlapping volumes in a common spaceactually pertain to the same source object, in which case the two videoobjects can be represented in the output video simultaneously.

In some embodiments, selected source objects can be depicted in discreteoutput videos that are displayed on respective display devices asopposed to conventional video synopses where all output video objectsare depicted in a common video sequence. An advantage of showing thepassage through space and time of a selected source object in separatevideo sequences is that the video objects in each sequence are imaged atdifferent times and are therefore necessarily spatially separated. Thismay not be, and typically is not, the case when a common source objectis images by two or more cameras having overlapping fields of view inwhich case care must be taken to ensure that two or more video objectsbelonging to the same object do not mutually obscure each other in theoutput video.

In one aspect, the present invention proposes an extension of VideoSynopsis which displays the detected objects from multiple cameras, ortheir representations, on a reference model such as a 3D model or a 2Dmap, instead of displaying the objects on the original background imageof the recorded scene. The proposed model-based Video Synopsis allowsdisplaying objects viewed by different cameras, either static ordynamic, as long as it is possible to map the object locations from thecamera plane into the reference model. In addition to displaying thecropped or warped pixels of the original objects, icons canalternatively be used. This option can overcome the limitations ofinaccurate segmentation and geometric distortion.

The main challenges for providing a model based Video Synopsis are (1)successful detection of the scene objects, (2) successful multipleobject tracking and (3) correct mapping of the object trajectories intoa given reference model. The mapped object trajectories can bere-arranged in time in a similar manner as described in [1, 3], andpresented by graphical elements moving relative to the reference model.

An additional method described in this invention is synchronized videosynopsis, which is applicable to videos from multiple stationarycameras. A set of synopsis videos is generated, one per camera, whichsynchronize the display of objects viewed by more than a single camera.In other words, if an object is seen by different cameras at the sametime, then it is displayed simultaneously in all the correspondingsynopsis videos. This is an improvement over known video synopsismethods, which process the video from each camera independently asdistinct from previous video summarization methods which display thedifferent views of the same object at different times. When an object isviewed concurrently by several cameras, independent processing willresult in the same objects being displayed at a different time in eachsynopsis video. The novelty of the proposed synchronized video synopsisis based on joint spatio-temporal rearrangement, which extends thetechnique described in [1, 2, 3, 4] from a single space-time volume intomultiple constrained volumes, as described in detail below.

In another aspect, the invention provides a method that generates asingle Video Synopsis from multiple input videos with no need of areference model or advanced spatio-temporal rearrangement scheme. Thissimple method assigns new display times to all the extracted objects ina common space time volume and renders them over an arbitrary backgroundimage. An extension is also proposed for the case in which an objectre-identification method is available which allows to connect betweenmultiple recordings of the same real object at different times or bydifferent cameras.

Embodiments of the invention thus relate to a number of different cases.In one case, multiple cameras record video sequences of a site fromdifferent camera perspectives and an output video is created thatdisplays representations of objects, such that object instances imagedby different cameras at different times are represented simultaneouslyin the output video, thus creating a video that is shorter in durationthat the cumulative durations of the component video sequences. For atleast one location in the output video, there are represented instancesimaged by two different cameras. This precludes the trivial cases ofconcatenating multiple video sequences while omitting some frames or ofconcatenating two video synopses side by side or of displaying two videosynopses sequentially or of fast forward with the same speed for all thesource videos, which trivially preserves the same display times forobjects which have been viewed simultaneously in the source videos.

In another case, not all instances of the same object are recordedsimultaneously by all cameras but the different video sequences aresynchronized to determine which object instances are common to two ormore cameras. These are then represented simultaneously in the outputvideo, thereby showing in a video of shorter duration the spatialprogress of the object as it traverses the site.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to understand the invention and to see how it may be carriedout in practice, embodiments will now be described, by way ofnon-limiting example only, with reference to the accompanying drawings,in which:

FIG. 1 is a flow diagram of a model-based video synopsis systemaccording to an embodiment of the invention;

FIG. 2 is a block diagram showing the functionality of the videoanalysis stage;

FIG. 3 is a pictorial representation of the object extraction andmapping pipeline;

FIG. 4 is a block diagram showing functionality of the generation of amodel based video synopsis;

FIG. 5 is a pictorial representation of the spatio-temporalrearrangement (retiming) process; and

FIG. 6 is a block diagram showing the functionality of the synchronizedvideo synopsis pipeline.

DETAILED DESCRIPTION OF EMBODIMENTS

In the following description of some embodiments, identical componentsthat appear in more than one figure or that share similar functionalitywill be referenced by identical reference symbols.

FIG. 1 is a flow diagram of a model-based video synopsis system 10according to an embodiment of the invention. The system input ismultiple video sequences 11 which are passed into a video analysis stage12 in which the scene objects are detected, tracked and mapped into areference model as described in greater detail below. The next stage 13performs spatio-temporal rearrangement of the objects in the referencemodel domain as described in greater detail below. The system output ismodel based video synopsis 14 which displays the objects or theirrepresentations asynchronously in the reference model.

The system 10 is implemented as follows. We assume that we are given areference model constituting a predetermined space such as a 2D map or a3D point cloud which represents a site in the real world. A singlecamera or multiple cameras, either static or dynamic, are viewingregions in the site represented by the reference model. The generatedVideo Synopsis is a video clip which renders the objects as seen in thesingle or multiple cameras, or their graphical representations, movingin the site model. The objects can be reordered in time as in normalvideo synopsis.

As shown by FIG. 1, the proposed pipeline consists of two main stages:video analysis and video synopsis generation. The video analysis stagegets as input multiple video sequences from different sources, extractsthe objects viewed in these videos and represents their trajectories ina common reference model. This representation is the input of the videosynopsis generation stage, which performs spatio-temporal reordering andrenders a model-based video synopsis. The multiple input video sequencesare typically recorded by different cameras. The extracted objects areanalyzed and their trajectories are mapped into a common referencemodel. In case of multiple cameras, this process is performedindependently for each camera. Then an additional step is applied whichmerges trajectories extracted from different cameras which represent thesame real object.

FIG. 2 is a block diagram showing in more detail the functionality ofthe video analysis stage 12 shown in FIG. 1. Given video streams 21recorded by multiple cameras 22, we extract the objects recorded by eachcamera independently. The object extraction process described in greaterdetail below is performed by the following modules: an object detectionmodule 23 which detects objects per frame and provides 2D boundingboxes; an object tracking module 24 which connects the detected boundingboxes into tracklets; and a mapping module 25 which transform thetracked object bounding boxes from the 2D plane of the original videointo a common reference model which is either 2D or 3D. Following thatmapping, an object merge module 26 merges tracklets from differentcameras representing the same real object. For each processed object, agraphical representation is generated 27. This representation is usedfor rendering the object in a model based video synopsis.

FIG. 3 is an illustration of the proposed object extraction and mappingpipeline. At stage (a) objects recorded by each camera are detected andtracked independently. At stage (b) the tracked objects are mapped intoa common reference model, either 2D or 3D. At stage (c) different viewsof the same real object in the scene are merged based on overlap andmotion similarity in the reference model coordinates and stage (d)derives the graphical representation of the object in the referencemodel.

Object Extraction (OX)

While some OX methods, like background subtraction, can mostly beapplied to static cameras, other OX methods can be applied to dynamiccameras as well. This is unlike classic Video Synopsis that is limitedto stationary cameras since it requires the separation betweenforeground and background pixels.

Our proposed OX pipeline consists of two steps. In the first step, the2D object detector 23, as described above is applied to video frames,giving locations of detected objects (e.g. objects masks or boundingboxes) at the image plane. A corresponding class label is possible foreach object.

In the second step, a tracking method is applied to connect the singleframe detections into multi-frame objects, each multi-frame object beingrepresented by a sequence of 2D detections. Pixel level masks of thedetected objects can be generated, if required, using instancesegmentation methods.

Optionally, an additional object filtering step can be performed basedon analysis of the tracked object trajectories and/or the segmentedpixel level masks. Information about the object properties can beachieved by extracting different features describing the object class,color distribution, size, motion etc. Similarly to [6], the relevance ofeach object to some predefined objective function can be calculated.This makes it possible to map into the reference model just a filteredsubset of objects, which meet one or more predetermined filteringcriteria based on their respective objective function values e.g. onlyhumans or only large objects. In such a case, the final video synopsisis shorter and displays just these relevant objects. The objects canalso be reordered in time by their relevance to the objective function.

Mapping into the Reference Model

In order to display the objects at their correct locations in thereference model, the source cameras are calibrated with the referencemodel in known manner as described above. For each object i and camerac, an object trajectory is defined as:

O _(i) ^(c)={(x _(j) ,y _(j) ,t _(j))}_(j=0) ^(J)  (1)

where (x_(j),y_(j)) is some designated predefined point in the object attime t_(j) (e.g. the bottom middle point of the 2D object bounding box).The trajectories of the objects which have been detected and tracked ineach camera are then mapped into the reference model:

M(O _(i) ^(c)))={(M(x _(j) ,y _(j)),t _(j))}_(j=0) ^(J)  (2)

where M(·) is a mapping function from the 2D camera plane into thereference model. Each point M(x_(j),y_(j)) represents the objectlocation in the reference model at time t_(j). This location can be, forexample, the bottom middle point of the object's 2D bounding rectangle,or the bottom center point of its 3D bounding box, depending on thereference model dimension.

Based on the mapped object trajectory, a spatio-temporal tube isconstructed which consists of the areas (or volumes) of the objectrepresentations along time:

Tube_(i) ={B _(j) ,t _(j)}_(j=0) ^(J)  (3)

Here, B_(j) are defined as the 2D or 3D regions containing the objectrepresentations in all times t_(j).

Object Tube from Multiple Cameras

If an object i is viewed from multiple cameras at the same time, then ithas multiple overlapping trajectories in the reference model's domain.For each camera pair {c,d}, the trajectory pair M(O_(i) ^(c)),M(O_(i)^(d)) can be matched by area (or volume) overlap in corresponding framesand by motion similarity. It is possible to merge the matching objecttrajectories into a single trajectory M(O_(i)).

An example of a merged trajectory consists of all the mapped locationsM(x_(j),y_(j)) in times t_(j) belonging to the original trajectoriesM(O_(i) ^(c)),M(O_(i) ^(d)). In case of overlapping times t_(j), when anobject is seen in both cameras, the mapped locations in bothtrajectories can be averaged.

The process of matching and merging object trajectories can be repeatedmany times until all the trajectories which represent the real object iand have partial time overlap are merged into a single object trajectoryM(O_(i)).

As mentioned above, non-overlapping instances of the same object, asdetected by object re-identification methods, either from the samecamera or from different cameras, can also be merged. An example of suchmerge is concatenation of the mapped locations M(x_(j),y_(j)) from bothtrajectories. Merging objects in the temporal domain can be done in manyways, for example taking the original times t_(j), or shifting the timesof the later trajectory so that it starts immediately after the end ofthe first trajectory.

As in the case of a trajectory from a single camera, a merged trajectoryis also converted into a spatio-temporal tube (Eq. 3).

Graphical Object Representation

In a model based Video Synopsis the rendered objects may be graphicallyrepresented in different ways depending on the application and thereference model type (2D, 3D etc.). The possible graphicalrepresentations include the original object pixels or objecttransformation outputs: object thumbnails, head or face thumbnails,icons, synthetic 3D models such as 3D meshes, or any other graphicalrepresentation.

Object representation by its original pixels requires warping the objectpixels by the appropriate geometric transformation from the cameracoordinates to the reference model. In many cases, this can distort thedisplayed object significantly. Such distortions can be avoided whenusing the other mentioned graphical representations.

If an icon or a synthetic graphical model is selected to represent theobject, different object attributes can be presented such as objectclass, color, size, direction, pose etc. Alternatively, color coding canbe used which indicates the displayed object relevance to specifiedcriteria such as appearance time, dwell time, similarity to a queryobject etc.

It should be noticed that the object graphical representation may betwo-dimensional or three-dimensional, depending on dimension of thereference model. This affects the temporal re-arrangement as describedbelow.

Video Synopsis Generation

The second stage of the proposed method generates a model-based videosynopsis based on the object representation by spatio-temporal tubes, asdescribed above with reference to (Eq. 3). FIG. 4 is a block diagramshowing functionality of the generation of a model based video synopsisunit 400, whose input is a set of video objects represented asspatio-temporal tubes 41 (Eq. 3). Each tube consists of a sequence of 2Dor 3D bounding boxes B_(j) in the reference model coordinates withcorresponding times t_(j) 41 a, 41 b. A spatio-temporal rearrangement ofthese tubes is performed by a retiming module 42, which shifts all thetime values of each object i by a constant Δ_(I) to form re-timed tubes43 a, 43 b constituting output objects. The set of the retimed tubes 43a . . . k is passed to a rendering module 44 that generates amodel-based video synopsis as explained below. In the case of a 3Dreference model, the rendering module 44 inputs also a user defined 3Dviewpoint 45 for the generation of 2D video frames from the 3D referencemodel and 3D objects.

FIG. 5 is a pictorial representation of the spatio-temporalrearrangement (retiming) process. Stage (a) shows the locations ofobject i after mapping into the reference model in the original times t₁^(i) . . . t₃ ^(i). Stage (b) shows the locations of an object j inoriginal times t₁ ^(j) . . . t₄ ^(j), and stage (c) shows the results ofthe spatio-temporal rearrangement by the retiming process describedabove with reference to FIG. 4. The original times are shifted by objectdependent constants Δ_(i) and Δ_(j), to obtain a synopsis video showingall objects but of shorter review time compared to the original video.

Spatio-Temporal Rearrangement

After extracting the scene objects and mapping them into the referencemodel, we have a set of spatio-temporal tubes, {Tube_(i)}_(i=1) ^(n). Inorder to achieve a dense short video, a retiming step is performed inwhich the tubes are re-arranged in time by adding an offset Δ_(i) to therendering times of each tube i. Δ_(i) can be either positive ornegative.

Retimed−Tube_(i) ={B _(j) ,t _(j)+Δ_(i)}_(j=0) ^(J)  (4)

As described in [1, 3], a short video synopsis can be achieved using agreedy retiming algorithm. According to one approach, the objects canfirst be sorted by their total size, Σ_(j)|B_(j)|, which is the volumeof the 3D tube along all frames. Alternatively, the objects can besorted by their relevance to a predefined function [6], based on objectinformation which was calculated during the object extraction step(Section 3.1.1). After sorting, for each tube i, a 1D parameter searchis performed along the time axis for finding the first time {tilde over(t)} in which the amount of spatial overlap with already located tubesis smaller than a threshold. The retiming offset is then determined asΔ_(i)={tilde over (t)}−t₀.

The original retiming algorithm in [1, 3] was described in the contextof a 3D space-time domain with time axis and two spatial axes. This canbe applied directly to any 2D reference model such as a map or aerialimage, and set of tubes consisting of 2D object representations.Applying the same retiming algorithm in a 4D domain with time axis andthree spatial axes is straightforward. The search for optimal retimingoffsets Δ_(i), is still done along the time axis, and the onlydifference is the calculation of tubes' overlap which is done onintersections of 3D boxes instead of 2D rectangles. Therefore, nearoptimal retiming can be achieved using the same greedy approach.

Video Synopsis Rendering

The final model-based Video Synopsis is a synthetic video created byrendering the retimed tubes on a background image generated by thereference model.

In the case of a 2D reference model, such as a map, a video frame ofrendering time t is created by displaying all the representations ofobjects with shifted times t_(j)=t in the corresponding locations B_(j)over a background image generated by the reference model, such as themap itself or a combination of the background images projected onto thereference model plane. The term “location” refers to the entire 2D or 3Dregion containing the object representation at rendering time t_(j)=t,not to a single 2D (or 3D) point.

In the case of a 3D reference model, a 3D viewpoint should be firstdetermined, either fixed viewpoint for the entire clip, or varyingviewpoint as a function of the rendering time t. This defines an imageplane relative to the reference model. The rendered frame's backgroundis generated by projection of the 3D model over the selected imageplane. Then the object representations are displayed on the projectedbackground in the corresponding locations (and poses, if therepresentations are 3D synthetic models).

Synchronized Video Synopsis

This section describes synchronized Video Synopsis (SVS). Given are Kvideo streams recorded by stationary cameras, typically with full orpartial overlaps between their fields of view. In the most relevantscenario for SVS, there are pairwise overlaps between some of thecameras' field of views. Otherwise, SVS works properly but the result isidentical to the original video synopsis. We assume that the frame timesin each video are known, so that the original videos can besynchronized. Our goal is to generate K synchronized video synopses, sothat if an object in the scene is seen at the same time t in twooriginal videos i and j, then it will be displayed in the correspondingvideo synopses at the same time {tilde over (t)}.

In the following we describe the SVS pipeline step by step.

FIG. 6 is a block diagram a block diagram showing the functionality ofthe of the synchronized video synopsis pipeline 60. Multiple cameras 61independently record respective video streams 62 whose objects areextracted. Similarly to model-based video synopsis, the objectextraction process is performed by an object detection module 63 whichdetects objects per frame and provides 2D bounding boxes and an objecttracking module 64 which connects the detected bounding boxes intoobject trajectories. A trajectory matching module 65 matches objecttrajectories from multiple cameras based on their appearance similarityso that sets of trajectories are created which represent simultaneousviews of the same real object by different cameras. The objecttrajectories are then rearranged by a joint retiming module 66 in Kspatio-temporal domains, so that an equal time offset Δ_(m) is assignedto all of the trajectories belonging to set m. The retimed objects fromeach original video are passed to a rendering module 67 which generatesmultiple synchronized video synopses 68. Any object which was recordedat the same original time by multiple cameras is displayedsimultaneously by the resulting video synopses.

Object Extraction

As a first step of the SVS pipeline, object extraction is performed foreach video independently as described above. This results in a set ofobject trajectories Objs(c)={(O_(i) ^(c)}_(i=1) ^(n) ^(c) where {O_(i)^(c)} are object trajectories with trajectory index i and camera indexc, as defined by Eqn. 1, and the total number of trajectories per camerac is n_(c). Unlike model-based video synopsis, no mapping into areference model is required, since multiple video synopses are generatedin the coordinate systems of the original videos. On the other hand, asin model based video synopsis, it is possible to analyze the extractedobjects and filter them according to one or more predefined filteringcriteria. In such case, the steps described below are performed for thetrajectories of only those objects which match the filtering criteria.

Grouping Objects Across Cameras

The second step of the SVS pipeline is grouping video objecttrajectories into disjoint sets of related objects in all the videosequences. Two video objects are defined as related if they representthe same real object. The grouping process consists of two stages: firstwe find pairs of related objects, and then we group them into disjointsets using a graph based algorithm.

Matching Object Pairs

As mentioned above under the heading “Object Matching”, it is possibleto match video object pairs by applying different similarity functionswhich output different results. If we are interested to group all theinstances of the same object in different times, an objectreidentification method is applied which compares video object pairs anddetermines whether or not they represent the same real object.Alternatively, we may be interested just in finding synchronized viewsat the same time by different cameras. In such a case video object pairswith overlapping time ranges should be compared, and (optionally)different similarity methods can be also used such as space overlapand/or motion similarity.

Grouping Related Objects After comparing all the required video objectpairs, a undirected graph G is constructed which consists of vertices Vrepresenting the object trajectories O_(i) ^(c), and edges E betweenvertices (O_(i) ^(c), O_(i′) ^(c′)) if the similarity score between thecorresponding trajectories is greater than a threshold t_(sim). Theconnected components of G represent sets of related video objects, i.e.,all the objects at the same connected component are different views ofthe same real object. It should be noted that if only video objects withtime overlap have been matched, then there might be multiple sets ofrelated video objects which represent the same real object at differenttimes.

We denote these disjoint sets by

ID_(m) = {O_(i₁)^(c₁)  …  O_(i_(n_(m)))^(c_(n_(m)))}

where n_(m) is the number of trajectories in ID_(m), i_(j) is a videoobject index, and c_(j) is a camera index.

Joint Spatio-Temporal Rearrangement

The original video synopsis method re-arranges the extracted objects ina spatio-temporal domain whose coordinates system consists of the timeaxis and the spatial image axes. In the following we extend thespatio-temporal rearrangement (retiming) technique described in [1, 2,3, 4] into multiple spacetime volumes ST₁ . . . ST_(K) with spatialcoordinates corresponding to those of the input video sequences, and acommon time axis.

Similar as described above under the heading “Spatio-TemporalRearrangement”, the object trajectories (O_(i) ^(c)) are transformedinto spatio-temporal tubes {Tube_(i) ^(c)}_(i=1) ^(n) ^(c) , which areshifted along the time dimension by adding an offset Δ^(c) _(i) to theirrendering times. The same greedy algorithm is applied for finding theoffsets which best compress the objects in each spacetime volume ST_(c),but with the following constraint on the members of related object sets:

-   -   Let

O_(i₁)^(c₁)  …  O_(i_(n_(m)))^(c_(n_(m)))

be the trajectories of objects belonging to the mth set of related videoobjects, ID_(m), with start and end times {(s_(j), e_(j)}_(j=1) ^(n)^(m) , so that s₁≤s₂ . . . ≤s_(n) _(m) (i.e. the objects are sorted bytheir start times).

-   -   For each object O_(i) _(j) ^(c) ^(j) , j>1 define relative time        offset w.r.t the previous object, δ_(j)=min(s_(j),        e_(j−1))−s_(j−1).    -   Let Δ_(m) be the display time of O₁ ^(c) ¹ , the object with        smallest start time in ID_(m). Then the display time of the jth        object is Δ_(m)+Σ_(k=2) ^(j)δ_(k).

The above constraint implies that related video objects with overlappingtime ranges are displayed simultaneously in all the resulting videosynopses, while time gaps between related video objects withnon-overlapping time ranges are eliminated. A single parameter Δ_(m)determines the display times of the entire video object set ID_(m).While searching for an optimal Δ_(m) value, the entire set ID_(m) isadded simultaneously to the multiple volumes ST₁ . . . ST_(K), and aone-dimensional search is performed along the common time axis. The costof each time coordinate t is determined by the sum of its costs in ST₁ .. . ST_(K).

Video Synopsis Rendering

The result of the above joint rearrangement process is K sets ofre-ordered (or retimed) tubes, from which K video synopses areconstructed using the same rendering method described in by the authorsof the original video synopsis [1, 2, 3, 4]. Due to the time shift byequal offsets Δ_(m) of each set of matching tubes, the resultingsynopses are synchronized so that the different views of an object atthe same original time are displayed simultaneously by all of theresulting video synopses. This is different from processing the videosindependently by the original method, which displays the different viewsin different times.

Alternative Joint Rearrangements

The above constraint on the display times of related object sets ID_(m)can be modified or replaced by alternative procedure, which results indifferent variants of joint spatio-temporal rearrangement. Here wedescribe one such alternative procedure:

-   -   Let c₁ . . . c_(k) be the camera indexes of objects belonging to        the mth set of related video objects, ID_(m).    -   For each subset of objects in ID_(m) with camera index c_(j),        compute relative display times

δ_(j₁) = 0 ≤ δ_(j₂)  … ≤ δ_(j_(n_(c_(j))))

by regular spatio-temporal rearrangement (retiming) in ST_(j) using thealgorithm descried in [1, 3]. The term n_(c) _(j) is the number ofobjects recorded by the jth camera.

-   -   Search for an optimal display time Δ_(m) which minimizes the sum        of costs of adding each object O_(j) _(l) ^(C) ^(j) to the        relevant space-time volumes ST_(j) in time Δ_(m)+δ_(j) _(l)

The above procedure applies the regular retiming algorithms multipletimes to determine the relative display times within each camera relatedsubset. Afterwards, the entire set ID_(m) is added simultaneously to themultiple volumes ST₁ . . . ST_(K) by a one-dimensional search along thecommon axis time, which determines the final object display times. Theresult is a set of video synopses in which all the instances of the samereal object are displayed simultaneously. Unlike the variant describedabove, if a real object is recorded by different input videos indifferent original times, its recordings are still shown simultaneouslyby the resulting video synopses.

Simple Combined Video Synopsis

In the previous description, we have proposed two advanced methods forthe generation of Video Synopsis from multiple cameras: model-based VSand synchronized VS. We now propose a simple complementary method. Givenmultiple videos from different cameras, either static or dynamic, theproposed method generates a single Video Synopsis which reduces theoverall review time. We first describe a basic procedure, than add anoptional extension based on object re-identification.

Basic Method Object Extraction

Given multiple input videos, the object extraction step is performed foreach input video independently by means of object detection,segmentation of pixel-level masks, multiple object tracking and(optionally) object filtering as described above (under the heading“Object Extraction (OX)”).

Spatio-Temporal Rearrangement

After extracting object trajectories from all input videos,spatio-temporal rearrangement (retiming) is applied in a commonspace-time volume ST. The spatial dimensions of ST are defined by theinput videos widths and heights, so that each object frame (i.e.bounding box and segmented mask) remains with its original spatiallocation in the common spatio-temporal domain. Also here, “location”refers to the entire 2D region of the segmented object mask, not only asingle 2D point. Since all the objects are mapped into the samespace-time volume, a known retiming algorithm such as described in [1,3] can be applied for assigning new display times so that the overallreview time is minimized subject to overlap cost.

Rendering

After determining the display times of all the objects, it isstraightforward to render the segmented object masks, or any othergraphical representation, over an arbitrary background image such asblack/checkerboard background, or a background image learnedstatistically from one of the input videos (as in [1, 3]), etc. Theresult is a single Video Synopsis that not only reduces the review timecompared to the source video, but also makes it possible to watch allthe relevant content in a single output video. All this is achieved withno need of camera calibration, object matching, mapping into referencemodel space and graphical rendering as in the model-based Video Synopsisdescribed previously.

Extension with Object Re-Identification

If an object re-identification method can be applied as described aboveunder the heading “Object Matching”, it is possible to find pairs ofrelated objects as described above under the heading “Matching ObjectPairs” and group them into related object sets as described above underthe heading “Grouping related Object”. Unlike synchronized VideoSynopsis, in which related objects are rendered simultaneously inmultiple output videos, the proposed method outputs a single VideoSynopsis. Therefore, the multiple input video objects are transformedinto a single output video object by the following steps:

-   -   1. Let ID_(m) be the mth set of related objects, O₁ . . . O_(n)        _(m) , with start and end times (s₁, e₁) . . . (s_(n) _(m) ,        e_(n) _(m) ) so that s₁≤s₂ . . . ≤s_(n) _(m) .    -   2. Set the relative offset of the first object δ₁=0.    -   3. For each object j>1, set the relative offset from the        previous object j₁ as δ_(j)=min(s_(j),e_(j−1))−e_(j). The offset        of the jth object from the first object is therefore ^(P)        _(k)=1^(j)δ_(k).    -   4. At that point, for each time s₁≤t≤^(P) _(k)=1^(n) ^(m)        δ_(ken) _(m) there exists one or more object with relevant        frame. The rendered object can be represented by the segmented        mask of one selected object frame (e.g. those with the smallest        index, or those with largest size), or by any linear combination        of multiple object frames.

The retiming and rendering steps as described above under the broadheading “Simple Combined Video Analysis” are now applied to thegenerated output video objects instead of the input video objects. Theresult is a single Video Synopsis that shows all the real objectsrecorded by the different input videos, where real objects recorded bymultiple cameras are represented by a single video object.

Without derogating from the above generalizations, the inventiveconcepts encompassed by the invention include the following:

Inventive concept 1: A computer-implemented method for generating anoutput video, the method comprising:

obtaining respective source videos recorded by at least two cameras in asite, each of said source videos comprising multiple video framescontaining video objects imaged by said cameras, said video objectscorresponding to multiple instances of one or more respective sourceobjects;

obtaining for detected video objects in each source video, respectivetracks containing locations of the respective video objects in the videoframes;

for at least some of said video objects computing output video objectshaving a new start display time of each video object

selecting a background image on which to render the retimed videoobjects; and

generating an output video by rendering the output video objects orgraphical representations thereof at their new display times over theselected background image such that:

-   -   (i) instances imaged by different cameras at different times are        represented simultaneously in the output video;    -   (ii) at least two output video objects originating from a common        camera have different relative display times to their respective        source objects; and    -   (iii) there exists at least one location in the output video        wherein there are represented instances imaged by two different        cameras.        Inventive concept 2: The method according to inventive concept        1, including after obtaining respective tracks and prior to        computing output video objects:

calculating locations of the detected video objects in a predeterminedspace at each frame time, based on said locations in the source videoand known parameters of the respective camera from which the frame isobtained.

Inventive concept 3: The method according to inventive concept 2,wherein displaying the output objects or graphical representationsthereof includes:

selecting a respective graphical representation for each object;

selecting a 3D viewpoint for each frame in the output video; and

displaying the respective graphical representations of the outputobjects inside the predetermined space by projecting graphicalrepresentations of the predetermined space and the objects onto a 2Dimage plane defined by the selected 3D viewpoint.

Inventive concept 4: The method according to any one of inventiveconcepts 1 to 3, wherein at least one of said videos is recorded by adynamic camera whose parameters are known at each frame time.Inventive concept 5: The method according to inventive concept 3,wherein selecting the 3D viewpoint is done before computing outputobjects, and computing output objects is performed only on objectsvisible in the selected 3D viewpoint.Inventive concept 6: The method according to any one of the precedinginventive concepts, wherein the graphical representation of the objectis a 3D mesh or 3D point cloud.Inventive concept 7: The method according to inventive concept 2 or anyinventive concept dependent thereon, wherein the predetermined space isa 2D map or a 2D diagram or a 2D satellite image, and the graphicalrepresentation of the object is a 2D icon.Inventive concept 8: The method according to inventive concept 2 or anyinventive concept dependent thereon, further including after calculatinglocations of the detected objects and before computing output objects:

matching recordings of each unique object by different cameras inoverlapping times based on appearance similarity methods and/orgeometric information such as overlapping locations and similar motionpattern in the predetermined space; and

merging sets of matching object tracks into a single object track in thepredetermined space by averaging respective matched locations of theobject at each overlapping frame time.

Inventive concept 9: A computer-implemented method for generating anoutput video, the method comprising:

obtaining respective source videos recorded by at least two cameras in asite, each of said source videos comprising multiple video framescontaining video objects imaged by said cameras, said video objectscorresponding to multiple instances of one or more respective sourceobjects traversing the site;

computing a background image for each source video;

computing respective tracks of video objects detected in each sourcevideo, wherein each track contains the locations of the respective videoobject at each video frame;

associating at least one set of related video objects which consists oftwo or more video objects from at least two different source videosrepresenting the same source object;

computing output objects each having a new display time, and

generating output videos for said at least two cameras by rendering theoutput objects at their new display times over the respective computedbackground image such that:

-   -   (i) for each camera there are at least two output objects, both        computed from video objects at said camera which do not appear        in any common frame at the source video, which are displayed in        the respective output video in at least one common frame;    -   (ii) for each camera there are at least two output objects, both        computed from video objects at said camera which appear in at        least one common frame at the source video, which have different        relative display times to their respective video objects; and    -   (iii) at least two related objects, each derived from a        different camera, are displayed in the respective output videos        at a common frame time.        Inventive concept 10: The method according to inventive concept        9, wherein there is an overlap between the fields of view of at        least two cameras, and two video objects in two source videos        are considered related if they represent an identical source        object viewed in at least one common frame time by the said        cameras.        Inventive concept 11: The method according to inventive concept        9 or 10, wherein computing output objects includes assigning a        new start display time to each object such that for every pair        of frames, two related objects having the same original time        have the same display time.        Inventive concept 12: The method according to inventive concept        1, wherein:

generating the output video comprises rendering the computed outputobjects over the selected background image, at the same spatiallocations as the respective video objects and at their new displaytimes.

Inventive concept 13: The method according to inventive concept 12,comprising:

associating at least one set of related video objects which consists oftwo or more video objects representing the same source object; and

merging each said set of related objects into a single video object, bymerging at each frame the instances of the respective video objects.

Inventive concept 14: The method according to inventive concept 13,wherein the merged object is constructed by selecting at each frame timethe pixel level segmentation of one of the objects in the set.Inventive concept 15: The method according to inventive concept 13,wherein the merged object is constructed by computing at each frame alinear combination of the pixel level segmentations of the objectsbelonging to the set.Inventive concept 16: The method according to any one of the precedinginventive concepts, including prior to computing output objects,filtering the objects according to at least one well-defined filteringcriterion, so that a reduced subset of objects is displayed by theoutput video.Inventive concept 17: The method according to inventive concept 16,wherein the at least one filtering criterion includes relevance to oneor more object attributes such as object class, duration, path, color,shape etc.Inventive concept 18: The method according to inventive concept 16 or17, wherein the retiming order of the selected objects is determined bytheir relevance to the at least one filtering criterion.Inventive concept 19: The method according to any one of the precedinginventive concepts wherein at least two objects are moving objects.Inventive concept 20: A computer-readable memory storing programinstructions, which when run on at least one processor cause the atleast one processor to implement the method according to any one of thepreceding inventive concepts.

It will also be understood that the system according to the inventionmay be a suitably programmed computer. Likewise, the inventioncontemplates a computer program being readable by a computer forexecuting the method of the invention. The invention furthercontemplates a machine-readable memory tangibly embodying a program ofinstructions executable by the machine for executing the method of theinvention.

It should also be noted that features that are described with referenceto one or more embodiments are described by way of example rather thanby way of limitation to those embodiments. Thus, unless stated otherwiseor unless particular combinations are clearly inadmissible, optionalfeatures that are described with reference to only some embodiments areassumed to be likewise applicable to all other embodiments also.

CONCLUSION

The invention proposes three methods and systems which extend VideoSynopsis from a single camera into multiple cameras.

First we proposed model based Video Synopsis, a synthetic video in whichobjects or their representatives are rendered asynchronously inside a 2Dor 3D reference model. This extends Video Synopsis from a cameradependent mode which is limited to a single stationary camera withspecific viewpoint, into model based mode in which objects viewed bymultiple cameras, static or dynamic, are displayed on a 2D or inside 3Dreference model. We described a system which extracts the scene objectsfrom multiple cameras, transforms them into representative tubes whichare located in the reference model, rearranges these tubes in a relatedspatio-temporal domain, and renders a model based Video Synopsis from aselected (fixed or dynamic) viewpoint. The resulting model based VideoSynopsis allows an efficient review of large video content inside aunified, camera independent scene model.

We also proposed synchronized video synopsis in which real objects whichare viewed by multiple cameras are displayed synchronously by multiplevideo synopses. This is based on object matching by similarity combinedwith a novel joint spatio-temporal rearrangement scheme. Synchronizedvideo synopsis is independent from scene reconstruction and cameracalibration, and significantly extends the capabilities of an automaticsearch in large video content.

Finally we proposed a simple method to display objects extracted frommultiple videos by a single Video Synopsis, with an extension torendering mode that utilizes information obtained by objectre-identification methods for joint display of multiple instances of thesame real object

1. A computer-implemented method for generating an output video, themethod comprising: obtaining respective source videos recorded by atleast two cameras in a site, each of said source videos comprisingmultiple video frames containing video objects imaged by said cameras,said video objects corresponding to multiple instances of one or morerespective source objects; obtaining for detected video objects in eachsource video, respective tracks containing locations of the respectivevideo objects in the video frames; for at least some of said videoobjects computing output video objects having a new start display timeof each video object selecting a background image on which to render theoutput video objects; and generating an output video by rendering theoutput video objects or graphical representations thereof at their newdisplay times over the selected background image such that: (i)instances imaged by different cameras at different times are representedsimultaneously in the output video; (ii) at least two output videoobjects originating from a common camera have different relative displaytimes to their respective source objects; and (iii) there exists atleast one location in the output video wherein there are representedinstances imaged by two different cameras.
 2. The method according toclaim 1, including after obtaining respective tracks and prior tocomputing output video objects: calculating locations of the detectedvideo objects in a predetermined space at each frame time, based on saidlocations in the source video and known parameters of the respectivecamera from which the frame is obtained.
 3. The method according toclaim 2, wherein displaying the output objects or graphicalrepresentations thereof includes: selecting a respective graphicalrepresentation for each object; selecting a 3D viewpoint for each framein the output video; and displaying the respective graphicalrepresentations of the output objects inside the predetermined space byprojecting graphical representations of the predetermined space and theobjects onto a 2D image plane defined by the selected 3D viewpoint. 4.The method according to claim 1, wherein at least one of said videos isrecorded by a dynamic camera whose parameters are known at each frametime.
 5. The method according to claim 3, wherein selecting the 3Dviewpoint is done before computing output objects, and computing outputobjects is performed only on objects visible in the selected 3Dviewpoint.
 6. The method according to claim 1, wherein the graphicalrepresentation of the object is a 3D mesh or 3D point cloud.
 7. Themethod according to claim 2, wherein the predetermined space is a 2D mapor a 2D diagram or a 2D satellite image, and the graphicalrepresentation of the object is a 2D icon.
 8. The method according toclaim 2, further including after calculating locations of the detectedobjects and before computing output objects: matching recordings of eachunique object by different cameras in overlapping times based onappearance similarity methods and/or geometric information such asoverlapping locations and similar motion pattern in the predeterminedspace; and merging sets of matching object tracks into a single objecttrack in the predetermined space by averaging respective matchedlocations of the object at each overlapping frame time.
 9. The methodaccording to claim 1, including prior to computing output objects,filtering the objects according to at least one filtering criterion, sothat a reduced subset of objects is displayed by the output video. 10.The method according to claim 9, wherein the at least one filteringcriterion includes relevance to one or more object attributes such asobject class, duration, path, color, shape etc.
 11. The method accordingto claim 9, wherein the output objects are displayed at respective timesthat are determined by a relevance of the output objects to the at leastone filtering criterion.
 12. The method according to claim 1, wherein atleast two objects are moving objects.
 13. A computer-implemented methodfor generating an output video, the method comprising: obtainingrespective source videos recorded by at least two cameras in a site,each of said source videos comprising multiple video frames containingvideo objects imaged by said cameras, said video objects correspondingto multiple instances of one or more respective source objectstraversing the site; computing a background image for each source video;computing respective tracks of video objects detected in each sourcevideo, wherein each track contains the locations of the respective videoobject at each video frame; associating at least one set of relatedvideo objects which consists of two or more video objects from at leasttwo different source videos representing the same source object;computing output objects each having a new display time, and generatingoutput videos for said at least two cameras by rendering the outputobjects at their new display times over the respective computedbackground image such that: (i) for each camera there are at least twooutput objects, both computed from video objects at said camera which donot appear in any common frame at the source video, which are displayedin the respective output video in at least one common frame; (ii) foreach camera there are at least two output objects, both computed fromvideo objects at said camera which appear in at least one common frameat the source video, which have different relative display times totheir respective video objects; and (iii) at least two related objects,each derived from a different camera, are displayed in the respectiveoutput videos at a common frame time.
 14. The method according to claim13, wherein there is an overlap between the fields of view of at leasttwo cameras, and two video objects in two source videos are consideredrelated if they represent an identical source object viewed in at leastone common frame time by the said cameras.
 15. The method according toclaim 13, wherein computing output objects includes assigning a newstart display time to each object such that for every pair of frames,two related objects having the same original time have the same displaytime.
 16. The method according to claim 1, wherein: generating theoutput video comprises rendering the computed output objects over theselected background image, at the same spatial locations as therespective video objects and at their new display times.
 17. The methodaccording to claim 16, comprising: associating at least one set ofrelated video objects which consists of two or more video objectsrepresenting the same source object; and merging each said set ofrelated objects into a single video object, by merging at each frame theinstances of the respective video objects.
 18. The method according toclaim 17, wherein the merged object is constructed by selecting at eachframe time the pixel level segmentation of one of the objects in theset.
 19. The method according to claim 17, wherein the merged object isconstructed by computing at each frame a linear combination of the pixellevel segmentations of the objects belonging to the set.
 20. The methodaccording to claim 13, including prior to computing output objects,filtering the objects according to at least one well-defined filteringcriterion, so that a reduced subset of objects is displayed by theoutput video.
 21. The method according to claim 20, wherein the at leastone filtering criterion includes relevance to one or more objectattributes such as object class, duration, path, color, shape etc. 22.The method according to claim 13, wherein the output objects aredisplayed at respective times that are determined by a relevance of theoutput objects to the at least one filtering criterion.
 23. The methodaccording to claim 13, wherein at least two objects are moving objects.